Java SDK: Register tasks as first-class TaskDef objects - #71057
Draft
jason810496 wants to merge 7 commits into
Draft
Java SDK: Register tasks as first-class TaskDef objects#71057jason810496 wants to merge 7 commits into
jason810496 wants to merge 7 commits into
Conversation
uranusjr
reviewed
Aug 4, 2026
| import org.apache.airflow.sdk.*; | ||
|
|
||
| @Builder.Dag(id = "sales_pipeline") | ||
| @Dag(dagId = "sales_pipeline") |
Member
There was a problem hiding this comment.
I prefer to keep the original naming
- The interface uses Dag and Task
- The Builder outer class (since the annotated class/method does not actually produce a dag/task)
- The argument being
id
The @task.stub TaskFlow support in providers-standard imports KNOWN_CONTEXT_KEYS, PlainXComArg, MappedOperator and the decorator base classes through the compat layer so the provider keeps working down to Airflow 2.11. Those symbols first ship in common-compat 1.19.0 (1.18.0 was released from main in the meantime without them), so the version is cut here for the standard provider's pin to resolve.
Stub tasks silently ignored TaskFlow call arguments, so a Dag author could not hand literals or upstream XCom results to a lang-SDK runtime. The decorator now binds the call to the stub's signature at parse time and captures an ordered arg spec (literal values and direct upstream XCom references, with pydantic-derived JSON value schemas) that serializes with the Dag, while rejecting what cannot cross the language boundary: custom XCom keys, aggregated mapped outputs, non-JSON literals, and stubs with arguments inside mapped task groups. Mapped (.expand()) stubs capture no spec and keep the legacy behavior until a follow-up delivers per-map-index bindings.
TIRunContext gains an arg_bindings field so a lang-SDK runtime receives the stub task's TaskFlow arg spec at startup. ti_run derives it from the serialized Dag only for stub operators, so regular tasks never pay for the lookup, and only for clients on the new API version -- gated on the Cadwyn VersionChangeWithSideEffects.is_applied check rather than a date comparison -- so stub Dags that predate arg bindings keep running against older clients, for which the version migration strips the field.
StartupDetails in the supervisor wire schema carries the new arg_bindings so foreign runtimes receive the spec at task startup, with a version migration that strips it for runtimes pinned to the previous schema. The Go and TS SDKs regenerate against the new schema version; the Go arg-binding runtime itself lands in a stacked follow-up PR.
An XComArg buried in a list or dict literal fell through to the JSON check, whose "pass it in its JSON form instead" advice is impossible to follow for a task output. Detect nested references up front and point the author at the working alternative: pass the upstream output as its own argument.
When a PR cuts a new provider version while the previous version is still being voted on, only the rcN tags exist on the apache remote - the final tag is pushed after the vote passes. The changes-table walk in _get_all_changes_for_package assumed every past version has a final tag and crashed with git exit 128 in that window, breaking CI for any PR that bumps a provider version during a release wave.
`dag.addTask("extract", Extract.class)` stored tasks as a plain
`Map<String, Class<out Task>>`, which leaves nowhere to hang anything
else a task needs: dependency edges, task-level configuration, and
argument wiring all have to attach to a per-task object, and a map of
classes cannot carry them. Introducing that object now keeps those
follow-ups additive instead of forcing another break of the registration
API later.
The annotation surface keeps `Builder.Dag` / `Builder.Task`, and the
interface users implement keeps the `Task` name, so the definition
objects are `DagDef` and `TaskDef` -- a pairing that stays unambiguous
next to `Task` at a use site. The SDK is pre-1.0, so the old
string-keyed overload is removed outright rather than deprecated.
jason810496
force-pushed
the
feature/java-sdk-taskflow-dsl
branch
from
August 5, 2026 15:40
3d77477 to
fecbcff
Compare
This was referenced Aug 5, 2026
Member
Author
I already separated them down into five. This PR depends on the other three PRs get merged first ( #69757, #71057, #71188) and contains their changes as well. I just added the "Diff for early review: " at the above of PR description now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Java SDK registered tasks as
dag.addTask("extract", Extract.class), backed by a plainMap<String, Class<out Task>>. That leaves nowhere to hang anything else a task needs — dependency edges, task-level configuration, and argument wiring all have to attach to a per-task object, and a map of classes cannot carry them. Introducing that object now keeps the follow-ups additive instead of forcing another break of the registration API later.How
TaskDef(its ID plus the class implementing it), registered withDagDef.addTask(TaskDef). ATaskDefbelongs to exactly oneDagDef, so registering the same instance twice — or with a second Dag — fails at Dag-parse time rather than silently sharing state.Builder.Dag/Builder.Task, and the interface users implement keeps theTaskname, so the definition objects areDagDefandTaskDef— a pairing that stays unambiguous next toTaskat a use site.addTask(id, Class)overload is removed outright rather than deprecated.What
Dag→DagDefand addTaskDef;Bundle/BundleBuildernow takeIterable<DagDef>.BuilderProcessoremitsdag.addTask(new TaskDef("...", T.class))from@Builder.Dag/@Builder.Taskclasses; the generated per-task inner classes still implementTask.Was generative AI tooling used to co-author this PR?